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ABSTRACT 



A method, apparatus, and article of manufacture for gener- 
ating statistics for use by a relational database management 
system. A global aggregate spool is generated for each of a 
plurality of partitions of a subject table that are spread across 
a plurality of processing units of a computer system. Each of 
the global aggregate spools is scanned to generate summary 
records. The summary records are then mergedlcTgeherate 
interval records for a compressed histogram of the subject 
table, wherein the compressed histogram includes both 
equal-height intervals and high-biased intervals. The com- 
pressed histogram can then be a nalyzed i to estimate the 
cardinality associated with one or lfnore search condition s of 
a uscrtfiTeTy or other SCjL statement. Compared to a strictly 
eqQa~£Eeight histogram, the compressed histogram allows 
the relational database management system to more accu- 
rately estimate the cardinality associated with various search 
conditions. As a result, the relational database management 
system can better optimize the execution of the user query. 

75 Claims, 2 Drawing Sheets 
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SELECTIVITY PREDICTION WITH 
COMPRESSED HISTOGRAMS IN A 
PARALLEL PROCESSING DATABASE 
SYSTEM 

5 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates in general to database management 
systems performed by computers, and in particular, to selec- 
tivity prediction with compressed histograms in a parallel 
processing database system. 

2. Description of Related Art 

Relational DataBase Management Systems (RDBMS) are 
well known in the art. In an RDBMS, all data is externally 15 
structured into tables. A table in an RDBMS is two 
dimensional, comprising rows and columns. Each column 
has a name, typically describing the type of data held in that 
column. As new data is added, more rows are inserted into 
the table. A user query selects some rows of the table by 20 
specifying clauses that qualify the rows to be retrieved based 
on the values in one or more of the columns. 

Structured Query Language (SQL) statements allow users 
to formulate relational operations on the tables. One of the 
most common SQL statements executed by an RDBMS is 25 
the SELECT statement. The SELECT statement generally 
comprises the format: "SELECT <clause> FROM <clause> 
WHERE <clause> GROUP BY <clause> HAVING <clause 
> ORDER BY <clause >" The clauses generally must 
follow this sequence, but only the SELECT and FROM 30 
clauses are required. 

Generally, the result of a SELECT statement is a subset of 
data retrieved by the RDBMS from one or more existing 
tables stored in the relational database, wherein the FROM 
clause identifies the name of the table or tables from which 35 
data is being selected. The subset of data is treated as a new 
table, termed the result table. 

The WHERE clause determines which rows should be 
returned in the result table. Generally, the WHERE clause 
contains a search condition that must be satisfied by each 40 
row returned in the result table: The rows that meet the 
search condition form an intermediate set, which is then 
processed further according to specifications in the SELECT 
clause. 

45 

The search condition typically comprises one or more 
predicates, each of which specify a comparison between two 
values comprising columns, constants or correlated values. 
Multiple predicates in the WHERE clause are themselves 
typically connected by Boolean operators. 5Q 

Database statistics can be used by the RDBMS to opti- 
mize the execution of the SQL statements. For example, 
statistics may be used to estimate cardinality for a selection 
of records based on the search condition specified for one or 
more of the columns of the records. Intuitively, the more 55 
intervals in the statistics for the column at issue, the higher 
the accuracy of the cardinality estimation. 

This can be argued from two extremes. At one extreme, 
there is only one interval. These "statistics" return an 
averaged number of rows per value as the estimate for all 60 
"simple selections" and this estimate remains the same for 
all values. This is the worst kind of statistics (though better 
than no statistics). 

At the other extreme, there exists one interval for each 
distinct value in the column. This provides the perfect 65 
cardinality "estimation," However, this approach is imprac- 
tical as the number of intervals can become arbitrarily large. 
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So, a trade-off must occur. And obviously, there should be as 
many intervals as possible, such that the frequencies within 
a single interval are more uniform and at the same time, the 
storage overhead must be acceptable. 

Therefore, it makes sense to group column values that 
have roughly equal frequency in the same equal-height 
interval. The smaller the variation among frequencies in the 
same interval, the better the cardinality estimation. In the 
extreme, of course, if there is only one value (thus one 
frequency in the interval, which provides a 100% accurate 
estimation for that particular value (which is called a loner). 
And, this is exactly the idea behind high-biased intervals. 

High-biased intervals store explicit column values and 
frequencies, so that a 100% estimation accuracy is obtained 
for these loners. Moreover, the rest of the column values can 
be made more uniform, if the column values with highest 
frequencies are removed from the equal-height intervals and 
put into high-biased ones. This way, not only do loners 
receive perfect estimation, but non-loners also benefit from 
increased uniformity. 

Using the above concepts, the present invention has 
developed various techniques for optimizing the perfor- 
mance of an RDBMS using improved database statistics. 
The specifics of these improvements are described in more 
detail hereinafter. 

SUMMARY OF THE INVENTION 

The present invention discloses a method, apparatus, and 
article of manufacture for generating statistics for use by a 
relational database management system. A global aggregate 
spool is generated for each of a plurality of partitions of a 
subject table that are spread across a plurality of processing 
units of a computer system. Each of the global aggregate 
spools is scanned to generate summary records. The sum- 
mary records are then merged to generate interval records 
for a compressed histogram of the subject table, wherein the 
compressed histogram includes both equal-height intervals 
and high-biased intervals. The compressed histogram can 
then be analyzed to estimate the cardinality associated with 
one or more search conditions of a user query or other SQL 
statement. Compared to a strictly equal-height histogram, 
the compressed histogram allows the relational database 
management system to more accurately estimate the cardi- 
nality associated with various search conditions. As a result, 
the relational database management system can better opti- 
mize the execution of the user query. 

An object of the present invention is to collect improved 
database statistics. Another object of the present invention is 
to improve the performance of relational database manage- 
ment systems using the improved statistics. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Referring now to the drawings in which like reference 
numbers represent corresponding parts throughout: 

FIG. 1 illustrates an exemplary software and hardware 
environment that could be used with the present invention; 

FIG. 2 is a flow chart illustrating the steps necessary for 
the interpretation and execution of SQL statements, or other 
user interactions, according to the preferred embodiment of 
the present invention; and 

FIG. 3 is a flowchart that illustrates the logic for gener- 
ating statistics for records stored in a subject table according 
to the preferred embodiment of the present invention. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENT 

In the following description of the preferred embodiment, 
reference is made to the accompanying drawings which 
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form a part hereof, and in which is shown byway of 
illustration a specific embodiment in which the invention 
may be practiced. It is to be understood that other embodi- 
ments may be utilized and structural changes may be made 
without departing from the scope of the present invention. 
Overview 

The preferred embodiment of the present invention gen- 
erates a new kind of database statistics, known as a com- 
pressed histogiam^for use by the RDBMS. The compressed 
histogram provides better estimates than an eq ual-Jfreij?ht 
histogram, because-hrgtPbiased intervaVkre^cruded m the 
compressed histogram. Compared to the equal-height 
histogram, the compressed histogram allows the RDBMS to 
more accurate ly estimate th e cardinality associ ated wi th 
various" s earch^ond itigjia>.As a result, the kbBMS can 
better optimizetbe Texgcj Uion of SQL stat ements. 
Definitions 

Following are definitions used throughout this specifica- 
tion. 

Interval records are defined by five fields: Values, Mode, 
ModeFreq, MaxVal, and Rows. The information stored 
in these fields depends on whether the Intervals are 
High-Biased Intervals or Equal-Height Intervals. 

The following table describes the High-Biased Intervals: 



Interval Type 



Field 



Definition 



High- Biased 


Values 


Representation for the 






number of loners in the 






interval. 








When the 


Then the 






interval 


Values field 






stores this 


is set to . . . 






many loners 








1 


-1 






2 


-2 




Mode 


Smaller loner. 






ModeFreq 


Number of rows having the 






modal value. 






MaxVal 


Maximum value for the 






interval. 








The field is 


When the 






defined as 


interval 






the . . . 


contains this 








many loners 






larger loner 


2 






modal value 


1 




Rows 


Number of rows having the 






MaxVal value. 





The following table describes the Equal-Height Intervals: 



Interval Type 


Field 


Definition 


Equal-Height 


"Values 


Total number of values for 
all the non-modal values in 
this interval. 




Mode 


Most frequent value in the 
interval. 




ModeFreq 


Number of rows having the 
modal value. 




MaxVal 


Maximum value covered by 
the interval. 




Rows 


Total number of rows for 
the non-modal values in the 
interval. 



10 



intervals have their Values field equal to or greater than 0. 
For a high-biased interval with its Values field equal to -1, 
a single value (called a loner) is stored in the Mode of the 
interval. If the Values field is equal to -2, then two loners are 
stored in the high-biased interval: a first loner in the Mode 
field and a second loner in MaxVal field. A count of rows is 
stored in the ModeFreq field for the first loner and is stored 
in the Rows field for the second loner. 
Loner — a distinct value that is stored in a high-biased 
interval. 

Equal-Height Histogram — an array of ordered equal- 
height intervals. 

Compressed Histogram — an array of intervals which 
comprises high-biased or equal-height intervals, or 
both. In the latter situation, high-biased intervals are 
ordered before the equal-height intervals. 

Global Interval Size — the average number of rows to be 
fitted in one interval. In one embodiment, this is set to 
be the total number of rows in the table divided by 100. 

Local Interval Size — the Global Interval Size divided by 
the number of processing units in the system. 

Environment 



25 



Note that high-biased intervals are characterized by hav- 
ing their Values field equal to -1 or -2. Equal- height 



FIG. 1 illustrates an exemplary hardware and software 
environment that could be used with the present invention. 
In the exemplary environment, a computer system 100 is 
comprised of one or more processing units (PUs) 102, also 
known as processors or nodes, which are interconnected by 
a network 104. Each of the PUs 102 is coupled to zero or 
more fixed and/or removable data storage units (DSUs) 106, 
such as disk drives, that store one or more relational data- 
bases. Further, each of the PUs 102 is coupled to zero or 
more data communications units (DCUs) 108, such as 
network interfaces, that communicate with one or more 
remote systems or devices. 

Operators of the computer system 100 typically use a 
workstation 110, terminal, computer, or other input device to 
interact with the computer system 100. This interaction 
generally comprises queries that conform to the Structured 
Query Language (SQL) standard, and invoke functions 
performed by Relational DataBase Management System 
(RDBMS) software executed by the system 100. 

In the preferred embodiment of the present invention, the 
RDBMS software comprises the Teradata® product offered 
by NCR Corporation, and includes one or more Parallel 
Database Extensions (PDEs) 112, Parsing Engines (PEs) 
114, and Access Module Processors (AMPs) 116. These 
components of the RDBMS software perform the functions 
necessary to implement the RDBMS and SQL standards, 
i.e., definition, compilation, interpretation, optimization, 
database access control, database retrieval, and database 
update. 

Work is divided among the PUs 102 in the system 100 by 
spreading the storage of a partitioned relational database 118 
managed by the RDBMS software across multiple AMPs 
116 and the DSUs 106 (which are managed by the AMPs 
116). Thus, a DSU 106 may store only a subset of rows that 
60 comprise a table in the partitioned database 118 and work is 
managed by the system 100 so that the task of operating on 
each subset of rows is performed by the AMP 116 managing 
the DSUs 106 that store the subset of rows. 
The PEs 114 handle communications, session control, 
65 optimization and query plan generation and control. The PEs 
114 fully parallelize all functions among the AMPs 116. As 
a result, the system of FIG. 1 applies a multiple instruction 
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stream, multiple data stream (MIMD) concurrent processing into an execution plan. Moreover, an Optimizer subsystem 

architecture to implement a relational database management of the PE 114 may transform or optimize the execution plan 

system 100. using database statistics generated in a manner described in 

Both the PEs 114 and AMPs 116 are known as "virtual more detail later in this specification, 

processors" or "vprocs" The vproc concept is accompUshed 5 Block 204 represents the PE 114 generating one or more 

by executing multiple threads or processes in a PU 102, « messages" fr om the execution pla n, wherein each' step 

wherein each thread or process is encapsulated within a „ a vL— — r- Auni1 ,l . , 

vproc. The vproc concept adds a level of abstraction mes^g^assigned to an AMP 116 that manages the desired 

between the multi-threading of a work unit and the physical ' e< *°' ds ' ^ mentl0ned above > the rows of the tables m the 

layout of the parallel processing computer system 100. in database 11? maybe parUtioned or otherwise distributed 

Moreover, when a PU 102 itself is comprised of a plurality 10 mukl P le ***** U6 ' so mat multl P le AMPs U6 can 

of processors or nodes, the vproc concept provides for work at . tbe samC time on , the data of a S lven table ' If a 

intra-node as well as the intercede parallelism. re 1 uest f for data 10 a sm $ c row ' the PE 114 transmits the 

-r. „ , i. • i_ *nn -i steps to the AMP 116 in which the data resides. If the request 

The vproc concept results in better system 100 availability ■ £ i*- i . . _ j „ 

%l ♦ j • i , mi , is tor multiple rows, then the steps are forwarded to all 

without undue programming overhead. The vprocs also 1S . awh o- *u • a * 7. *?o 

• a , f i * * * • .1 < 15 participating AMPs 116. Smce the tables in the database 118 

provide a degree of location transparency, in that vprocs u a a- * u . j + * * t_ 

. . ... ... . jj ... maybe partitioned or distributed" across the DS Us 16 of the 

communicate with each other using addresses that are vproc- AX ;„ f iJC , r r . 4 , „„ T 

n *u *u j -a r c AMPs 116, the workload of performing the SQL query can 

specific, rather than node-specific. Further, vprocs facilitate . . . ' Aim , , * T _ t ., r ~~~ " ' 

j j i_ "j * i i r - I . ■ / - . be balanced among AMPs 116 and DSUsT.6. 

redundancy by providing a level of isolation/abstraction & 

between the physical node 102 and the thread or process. 20 Block 204 also represents the PE 114 sending the step 

The result is increased system 100 utilization and fault messages to their assigned AMPs 116. 

tolerance. Block 206 represents the AMPs 116 performing the 

The system 100 does face the issue of how to divide a required data manipulation associated with the step mes- 

query or other unit of work into smaller sub-units, each of sa 8 es received from the PE 114, and then transmitting 

which can be assigned to an AMP 116. In the preferred 2 5 a PP ro P riate responses back to the PE 114. 

embodiment, data partitioning and repartitioning may be Block 208 represents the PE 114 then merging the 

performed, in order to enhance parallel processing across responses that come from the AMPs 116. 

multiple AMPs 116. For example, the data may be hash Block 210 represents the output or result table being 

partitioned, range partitioned, or not partitioned at all (i.e., generated, 

locally processed). Hash partitioning is a partitioning 30 

scheme in which a predefined hash function and map is used Operation of the Preferred Embodiment 

to assign records to AMPs 116, wherein the hashing function * \ A , r JLJ - *~t 

generates a hash "bucket" number and the hash bucket \> According to the preferred embodiment of the present^ V 

numbers are mapped to AMPs 116. Range partitioning is a a ° ew ^nd of database statistics known as a 

partitioning scheme in which each AMP 116 manages the 35 com P res ^ d hist0 g ram > are generated for use by the Opti- 

records falling within a range of values, wherein the entire ^subsystem of the PE 114 in optimizing ; an exe cuhon 

data set is divided into as many ranges as there are AMPs pla i^ "^pressed tetogram includes high-biased inter- 

me M rt ~ *u * • 1 A wt) ++c vals and/or equal-height intervals that allow the Optimizer 

116. No partitioning means that a single AMP 116 manages . r , ™ « ... 

all of the records & subsystem of the PE 114 to more ac curately cstumte^he 

Generally, the PDEs 112, PEs 114, and AMPs 116 are 40 <^ ^J^ e < i various , <affi S2BlB<Kgi-j 

tangibly embodied in and/or accessible from a device, ionj)^n. 

media, carrier, or signal, such as RAM, ROM, one or more Typically, the compressed histogram is independently 

of the DSUs 106, and/or a remote system or device com- S eiierated f ° r a specified subject table and then stored as a 

municating with the computer system 100 via one or more sm & c field of a row m a system table in the relational 

of the DCUs 108. The PDEs 112, PEs 114, and AMPs 116 45 database 118 for later use b Y lhe Optimizer subsystem of the 

each comprise logic and/or data which, when executed, PE 114 " ^ PE 114 fe ^sponsible for generating the 

invoked, and/or interpreted by the PUs 102 of the computer compressed histogram, using a sequence of collection steps 

system 100, cause the necessary steps or elements of the senUo and performed by the^AMPsJ16. In the preferred 

present invention to be performed. embodiment, there are two statistics collection steps. ^ 

Those skilled in the art will recognize that the exemplary 50 A first collection step is responsible for building a global *""] ? 

environment illustrated in FIG. 1 is not intended to limit the aggregatespool^and a sequence of summary records on each 

present invention. Indeed, those skilled in the art will AMP-S6-particjp^gjnjhe_statisUcs,coUectip^(i.e., on 

recognize that other alternative environments may be used each AMP 116 that manages a partitioiLof thejsubiect tabled, 

without departing from the scope of the present invention. In wherein multiple copies of the first collection step are 

addition, it should be understood that the present invention 55 exec uted simultaneously and in parallel by the AMPs 116. In 

may also apply to components other than those disclosed this manner, the global aggregate spool may be considered 

nerem . partitioned in the same manner as the subject table. ? 

p fenr n Each row_Qf the_ global a g gre gate^spooLiDcludes:-(l) a 

bxecution ot ^QL Queries disjtinctvajy^ 

FIG. 2 is a flowchart illustrating the steps necessary for 60 the_ nu mber of_ra ws_in^he-par-dtion-ef-the,subject^table 

the interpretation and execution of user queries or other SQL ha ving the distinct va lue. The global aggregate spool is 

statements according to the preferred embodiment of the considered global in the sense that a distinct value from the 

present invention. subject table can only be found on a single AMP 116, 

Block 200 represents SQL statements being accepted by because the subject table is partitioned across multiple 

the PE 114. 65 AMPs 116. 

Block 202 represents the SQL statements being trans- Summary records are constructed from the glob aggregate 

formed by a Compiler or Interpreter subsystem of the PE 114 s pool, wherein each summary record includes: (1) a sort key, 
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(2) a distinct value, and (3) the number of rows in the tion. Specifically, this flowchart illustrates the logic for 

partition of the subject table having the distinct value. These generating statistics for records stored in the subject table, 

summary records are produced by scanning the global Block 300 represents the subject , able being partitioned 

She^ 

the participating AMPs 116. 5 J^c^oftte PUs 102 maniges at least one putition 

After initializing a first current summary record with the °^ su ^j ect ta ^^ e * 

first row of the global aggregate spool, the AMP 116 loops Block 302 represents each of partitions of the subject 

to rea d all of the rows of the global aggregate spool; u pon table being scanned to generate global aggregate spools, 

completion, the logic terminates. As each row of the global Block 504 represents summary records being constructed 

aggregate spool is read, if the accumulated count is less than from each of the global aggregate spools and sent to the 

or equal to the Local Interval Size, then the current row is coordinator AMP 116 

merg ed into the summary record; o thenvise, the summary Block 30fi re te j nterval Records be - ted b 
record is sent to the coordinator AMP 116 and the next a»™ j <• • • 
summary record is initialized to the current row/ If, at the coordiaator AMP 116 and participating AMPs 116. 
anytime, the count of a row of the global aggregate spool is 15 Block 308 represents a compressed histogram for the 
greater than or equal to the Loner criteria, then the summary subject table being generated from the Interval Records, 
record's count field is set to (-1)* (row's count) and the wherein the compressed histogram includes both Equal- 
summary record is sent to the coordinator AMP 116. Height Intervals and/or High-Biased Intervals. 

The frequency of a newly retrieved row from the global Block 310. repr esents the compressed histogram _bejp g 

aggregate spool is checked to see if the row qualifies as a 20 analy zed to estimate jcardinality-ass ociated with one o r more 

loner. A loner is a distinct value satisfying the condition: search^cj^ndilions-QLa-user-query or other S QL statement. 

Note that the "two phases" of statistics represented by 
/ s I Blocks 308 and 310, i.e., collecting statistics and using 
L statistics, are independent of one another. Usually, the sta- 
25 tistics are collected on various fields by explicitly issuing 
where f=frequency of the loner, T is the total number of rows "collect statistics" statements. Thereafter, the collected sta- 
in the table, and L is the maximum number of loners (e.g., tistics are used by the Optimizer subsystem of the PE 114 
200). If the row qualifies as a loner, then the summary record when processing a query. The arrow between Blocks 308 
is generated and sent directly to the coordinator AMP 116, and 310 are not meant to imply that, in order to use the 
without the summary record being merged with any other 30 statistics, the statistics have to be collected for every query, 
records prior to being sent to the coordinator AMP 116. The only "dependency" between these Blocks is that the 

Conceptually, the summary records can be viewed as a Optimizer subsystem cannot use the statistics or 

second level aggregation on the global aggregate spool. This optimization, if they do not exist (i.e., have not been 

extra level of aggregation is necessary, because the sum of collected) 

all the global aggregate spools from all of the AMPs 116 35 

may be too large to be accommodated by a single AMP 116. Conclusion 

With the two level aggregation, the maximum number of t j 1. j • • ex. * lj- 

summary records sent to the coordinator AMP 116 is 71115 concludes the description of the preferred embodi- 

approximately 100*(Number of AMPs 116). ment of the invention. The following paragraphs describe 

After receiving the summary records from all participat- 40 alternative embodiments for accomplishing the same 

ing AMPs 116, the second collection step executed by the invention. 

coordinator AMP 116 generates the Interval Records in In one alternative embodiment, any type of computer, 

conjunction with the participating AMPs 116. The summary such as a mainframe, minicomputer, web sever, workstation, 

records are scanned twice: first for constructing the High- 0 r personal computer, could be used to implement the 

Biased Intervals, and then second for constructing the present invention. In addition, any DBMS or other program 

Equal-Height Intervals. The High-Biased Intervals are com- t hat performs similar functions could be used with the 

pletely specified at this time (i.e., all five fields are properly present invention 

set) while the Equal-Height Intervals are only partially T iL u ' . ... 4 4l _ . . - . 

initiahzed (i.e., only the MaxVal field is set). All of the , ano * er al ™ ve embodiment, the partitions of the 

Interval Records are then sent to all participating AMPs 116, ' able nc f no < b u e s P read across separate data storage 

which fill in the details for the Equal-Height Intervals, i.e., 50 devices. Instead, the partitions could be stored on one or a 

Mode, ModeFreq, MaxVal, and Rows, while ignoring the few data stora 6 c devices sm ? l V t0 minimize the amount of 

High-Biased Intervals. temporary data storage required at each of the steps of the 

After processing by the second collection step, the com- method, 

pressed histogram stores only a specified number of Interval In yet another alternative embodiment, the steps or logic 

Records. In the preferred embodiment, the maximum num- 55 could be performed by more or fewer processors, rather than 

ber of records is 100, although other embodiments may use the designated and other processors as described above. For 

different values. This final version of the compressed his- example, the steps could be performed simultaneously on a 

togram is then stored in the database 118 for later use by the single processor using a multi-tasking operating environ- 

Optimizer function of the PE 114. men t 

Thereafter, the Optimizer subsystem of the PE 114 uses fi0 In slun the t invention disclos6S a method 

be compressed hologram to .provide cardinahly ^formation an(J ^ f manufacture for generating statis . 

for relations. Cardinality is the number of rows per AMP 116 . rr - , * * * 

that are selected from a relation satisfying conditions in a "« for b \ a relat !°. nal data ^f f management sys em A 

WHERE clause global aggregate spool is generated for each of a plurality of 

partitions of a subject table that are spread across a plurality 

Logic of the Preferred Embodiment 65 0 f processing units of a computer system. Each of the global 

FIG. 3 is a flowchart that illustrates the logic performed aggregate spools is scanned to generate summary records, 

according to the preferred embodiment of the present inven- The summary records are then merged to generate interval 
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records for a compressed histogram of the subject table, 
wherein the compressed histogram includes both equal- 
height intervals and high-biased intervals. The compressed 
histogram can then be analyzed to estimate the cardinality 
associated with one or more search conditions of a user 5 
query or other SQL statement. Compared to a stricdy 
equal-height histogram, the compressed histogram allows 
the relational database management system to more accu- 
rately estimate the cardinality associated with various search 
conditions. As a result, the relational database management 1Q 
system can better optimize the execution of the user query 

The foregoing description of the preferred embodiment of 
the invention has been presented for the purposes of illus- 
tration and description. It is not intended to be exhaustive or 
to limit the invention to the precise form disclosed. Many 
modifications and variations are possible in light of the 15 
above teaching. It is intended that the scope of the invention 
be limited not by this detailed description, but rather by the 
claims appended hereto. 

What is claimed is: 

1. A method for generating statics for records stored in a 20 
subject table in a computer system, comprising: 

(a) generating a global aggregate spool for each of a 
plurality of partitions of a subject table stored on the 
computer system, wherein the partitions are stored 
across a plurality of processing units of the computer 25 
system; 

(b) constructing one or more summary records from each 
of the global aggregate spools; 

(c) generating one or more interval records from the 
summary records; 30 

(d) constructing a compressed histogram from the interval 
records, wherein the compressed histogram includes 
both equal-height intervals and high-biased intervals; 
and 

(e) analyzing the compressed histogram to estimate car- 35 
dinality associated with one or more search conditions. 

2. The method of claim 1, wherein the generating step (a) 
is performed simultaneously and in parallel by the process- 
ing units. 

3. The method of claim 1, wherein the constructing step 40 
(b) is performed simultaneously and in parallel by the 
process units. 

4. The method of claim 1, wherein the generating step (c) 
is performed by a coordinator processing unit. 

5. The method of claim 1, wherein the constructing step 45 
(d) is performed by simultaneously and in parallel by the 
participating processing units and a coordinator processing 
unit. 

6. The method of claim 1, wherein each of the interval 
records is comprised of one or more fields selected from a 50 
group comprising Values, Mode, ModeFreq, MaxVal, and 
Rows fields. 

7. The method of claim 6, wherein values stored in the 
fields identify intervals as either equal-height intervals or 
high -biased intervals. 55 

8. The method of claim 6, wherein the Values field in the 
high -biased interval represents a number of loners in the 
interval. 

9. The method of claim 8, wherein the Mode field in the 
high -biased interval represents a first loner for the interval 60 
when the Values field represents the number of loners in the 
interval. 

10. The method of claim 9, wherein the MaxVal field in 
the high-biased interval represents a second loner for the 
interval when the Mode field represents the first loner in the 65 
interval and the Values field represents the number of loners 

in the interval. 
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11. The method of claim 10, wherein the ModeFreq field 
in the high-biased interval represents a count of rows for the 
first loner. 

12. The method of claim 11, wherein the Rows field in the 
high-biased interval represents a count of rows for the 
second loner. 

13. The method of claim 6, wherein the Mode field in the 
high-biased interval represents a smaller loner. 

14. The method of claim 6, wherein the ModeFreq field in 
the high-biased interval represents a number of rows having 
a modal value. 

15. The method of claim 6, wherein the MaxVal field in 
the high-biased interval represents a maximum value for the 
interval. 

16. The method of claim 6, wherein the Values field in the 
equal-heights interval represents a total number of values for 
all non-modal values in the interval. 

17. The method of claim 6, wherein the Mode field in the 
equal-heights interval represents a most frequent value in the 
interval. 

18. The method of claim 6, wherein the ModeFreq field in 
the equal-heights interval represents a number of rows 
having a modal value. 

19. The method of claim 6, wherein the MaxVal field in 
the equal-heights interval represents a maximum value cov- 
ered by the interval. 

20. The method of claim 6, wherein the Rows field in the 
equal-heights interval represents a total number of rows for 
non-modal values in the interval. 

21. The method of claim 1, wherein the summary record 
includes both a distinct value from the subject table and a 
number of rows in the subject table having the distinct value. 

22. The method of claim 1, wherein a partition of the 
global aggregate spool resides on each of the processing 
units. 

23. The method of claim 22, wherein each row of the 
global aggregate spool includes both a distinct value from 
the subject table and a number of rows in the subject table 
having the distinct value. 

24. The method of claim 22, further comprising scanning 
the global aggregate spool to identify loners. 

25. The method of claim 24, wherein a loner is a distinct 
value satisfying the condition: 




where f-frequency of the loner, T is a total number of rows 
in the subject table, and L is a maximum number of loners. 

26. An apparatus for generating statistics for records 
stored in a subject table in a computer system, comprising: 

(a) a computer system; 

(b) logic, performed by the computer system, for: 

(1) generating a global aggregate spool for each of a 
plurality of partitions of a subject table stored on the 
computer system, wherein the partitions are stored 
across a plurality of processing units of the computer 
system; 

(2) constructing one or more summary records from 
each of the global aggregate spools; 

(3) generating one or more interval records from the 
summary records; 

(4) constructing a compressed histogram from the 
interval records, wherein the compressed histogram 
includes both equal-height intervals and high-biased 
intervals; and 

(5) analyzing the compressed histogram to estimate 
cardinality associated with one or more search con- 
ditions. 
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27. The apparatus of claim 26, wherein the logic for 
generating (1) is performed simultaneously and in parallel 
by the processing units. 

28. The apparatus of claim 26, wherein the logic for 
constructing (2) is performed simultaneously and in parallel 
by the processing units. 

29. The apparatus of claim 26, wherein the logic for 
generating (3) is performed by a coordinator processing unit. 

30. The apparatus of claim 26, wherein the logic for 
constructing (4) is performed by simultaneously and in 
parallel by the participating processing units and a coordi- 
nator processing unit. 

31. The apparatus of claim 26, wherein each of the 
interval records is comprised of one or more fields selected 
from a group comprising Values, Mode, ModeFreq, MaxVal, 
and Rows fields. 

32. The apparatus of claim 31, wherein values stored in 
the fields identify intervals as either equal-height intervals or 
high-biased intervals. 

33. The apparatus of claim 31, wherein the Values field in 
the high-biased interval represents a number of loners in the 
interval. 

34. The apparatus of claim 33, wherein the Mode field in 
the high-biased interval represents a first loner for the 
interval when the Values field represents the number of 
loners in the interval. 

35. The apparatus of claim 34, wherein the MaxVal field 
in the high-biased interval represents a second loner for the 
interval when the Mode field represents the first loner in the 
interval and the Values field represents the number of loners 
in the interval. 

36. The apparatus of claim 35, wherein the ModeFreq 
field in the high-biased interval represents a count of rows 
for the first loner. 

37. The apparatus of claim 36, wherein the Rows field in 
the high-biased interval represents a count of rows for the 
second loner. 

38. The apparatus of claim 31, wherein the Mode field in 
the high-biased interval represents a smaller loner. 

39. The apparatus of claim 31, wherein the ModeFreq 
field in the high-biased interval represents a number of rows 
having a modal value. 

40. The apparatus of claim 31, wherein the MaxVal field 
in the high-biased interval represents a maximum value for 
the interval. 

41. The apparatus of claim 31, wherein the Values field in 
equal-heights interval represents a total number of values for 
all non-modal values in the interval. 

42. The apparatus of claim 31, wherein the Mode field in 
the equal-heights interval represents a most frequent value in 
the interval. 

43. The apparatus of claim 31, wherein the ModeFreq 
field in the equal-heights interval represents a number of 
rows having a modal value. 

44. The apparatus of claim 31, wherein the MaxVal field 
in the equal-heights interval represents a maximum value 
covered by the interval. 

45. The apparatus of claim 31, wherein the Rows field in 
the equal-heights interval represents a total number of rows 
for non-modal values in the interval. 

46. The apparatus of claim 26, wherein the summary 
record includes both a distinct value from the subject table 
and a number of rows in the subject table having the distinct 
value. 

47. The apparatus of claim 26, wherein a partition of the 
global aggregate spool resides on each of the processing 
units. 
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48. The apparatus of claim 47, wherein each row of the 
global aggregate spool includes both a distinct value from 
the subject table and a number of rows in the subject table 
having the distinct value. 
5 49. The apparatus of claim 48, wherein the logic further 
comprises logic for scanning the global aggregate spool to 
identify loners. 

50. The apparatus of claim 49, wherein a loner is a distinct 
value satisfying the condition: 

10 

where f» frequency of the loner, T is a total number of rows 
15 in the subject table, and L is a maximum number of loners. 

51. An article of manufacture embodying logic for gen- 
erating statistics for records stored in a subject table in a 
computer system, comprising: 

(a) generating a global aggregate spool for each of a 
2 0 plurality of partitions of a subject table stored on the 

computer system, wherein the partitions are stored 
across a plurality of processing units of the computer 
system; 

(b) constructing one or more summary records from each 
2 5 of the global aggregate spools; 

(c) generating one or more interval records from the 
summary records; 

(d) constructing a compressed histogram from the interval 
records, wherein the compressed histogram includes 

30 both equal-height intervals and high-biased intervals; 
and 

(e) analyzing the compressed histogram to estimate car- 
dinality associated with one or more search conditions. 

52. The method of claim 51, wherein the generating step 
35 (a) is performed simultaneously and in parallel by the 

processing units. 

53. The method of claim 51, wherein the constructing step 

(b) is performed simultaneously and in parallel by the 
processing units. 

40 54. The method of claim 51, wherein the generating step 

(c) is performed by a coordinator processing unit. 

55. The method of claim 51, wherein the constructing step 

(d) is performed by simultaneously and in parallel by the 
participating processing units and a coordinator processing 

45 unit. 

56. The method of claim 51, wherein each of the interval 
records is comprised of one or more fields selected from a 
group comprising Values, Mode, ModeFreq, MaxVal, and 
Rows fields. 

50 57. The method of claim 56, wherein values stored in the 
fields identify intervals as either equal-height intervals or 
high-biased intervals. 

58. The method of claim 56, wherein the Values field in 
the high-biased interval represents a number of loners in the 

55 interval. 

59. The method of claim 58, wherein the Mode field in the 
high-biased interval represents a first loner for the interval 
when the Values field represents the number of loners in the 
interval. 

60 60. The method of claim 59, wherein the MaxVal field in 
the high-biased interval represents a second loner for the 
interval when the Mode field represents the first loner in the 
interval and the Values field represents the number of loners 
in the interval. 

65 61. The method of claim 60, wherein the ModeFreq field 
in the high-biased interval represents a count of rows for the 
first loner. 
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62. The method of claim 61, wherein the Rows field in the 
high-biased interval represents a count of rows for the 
second loner. 

63. The method of claim 56, wherein the Mode field in the 
high-biased interval represents a smaller loner. 

64. The method of claim 56, wherein the ModeFreq field 
in the high-biased interval represents a number of rows 
having a modal value. 

65. The method of claim 56, wherein the MaxVal field in 
the high-biased interval represents a maximum value for the 
interval. 

66. The method of claim 56, wherein the Values field in 
the equal-heights interval represents a total number of values 
for all non-modal values in the interval. 

67. The method of claim 56, wherein the Mode field in the 
equal-heights interval represents a most frequent value in the 
interval. 

68. The method of claim 56, wherein the ModeFreq field 
in the equal -heights interval represents a number of tows 
having a modal value. 

69. The method of claim 56, wherein the MaxVal field in 
the equal-heights interval represents a maximum value cov- 
ered by the interval. 

70. The method of claim 56, wherein the Rows field in the 
equal-heights interval represents a total number of rows for 
non-modal values in the interval. 
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71. The method of claim 51, wherein the summary record 
includes both a distinct value from the subject table and a 
number of rows in the subject table having the distinct value. 

72. The method of claim 51, wherein a partition of the 
global aggregate spool resides on each of the processing 
units. 

73. The method of claim 72, wherein each row of the 
global aggregate spool includes both a distinct value from 

10 the subject table and a number of rows in the subject table 
having the distinct value. 

74. The method of claim 72, further comprising scanning 
the global aggregate spool to identify loners. 

15 75. The method of claim 74, wherein a loner is a distinct 
value satisfying the condition: 




where f=frequency of the loner, T is a total number of rows 
in the subject table, and L is a maximum number of loners. 

25 

* * * * * 
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